CMOS image sensor using high frame rate with frame addition and movement compensation

ABSTRACT

A method of forming a composite image using a CMOS image sensor. The method comprises capturing a plurality of frames using the image sensor, identifying a reference point in each of the frames, and aligning the frames using the reference point. Finally, the frames are combines, such as be an arithmetic addition, into the composite image.

TECHNICAL FIELD OF THE INVENTION

The present invention relates generally to CMOS image sensors, and moreparticularly, to a method and apparatus for increasing the dynamic rangeand/or movement stability of the image sensor.

BACKGROUND OF THE INVENTION

Integrated circuit technology has revolutionized various fieldsincluding computers, control systems, telecommunications, and imaging.For example, in the imaging field, CMOS image sensors have proved to beless expensive to manufacture relative to CCD imaging devices. Further,for certain applications, CMOS devices are superior in performance. Forexample, the signal processing logic necessary can be integratedalongside the imaging circuitry, thus allowing for a single integratedchip to form a complete stand-alone imaging device. CMOS image sensorsare now available from a variety of manufacturers, including theassignee of the present invention, OmniVision Technologies, Inc.

The primary building block of an image formed by a CMOS image sensor isa pixel. The number, size and spacing of the pixels determine theresolution of the image generated by the imaging device. The pixels of aCMOS image sensor are semiconductor devices that transform incidentlight photons into current signals. The signal produced by each pixel isgenerally extremely small, but is related to the amount of incidentlight photons.

Optimally, the image sensor must operate in a myriad of light andmovement conditions. The image sensor when used in a mobile phoneapplication may be required to record images in low light conditions,since a flash is not readily available. In such a situation, theexposure time of the image sensor must be increased, much like for aconventional camera using photographic film. However, the increase inexposure time will give rise to blurring of the image if the sensor ismoved or shaken.

Further, in other applications, such as digital still cameras, thedynamic range of the image sensor may be inadequate to compensate for animage that has both low light sections and high light sections. In otherwords, shadows and light are not adequately resolved. Dynamic range isgenerally defined as the highest possible unsaturated signal divided bythe noise floor of the pixel. Thus, dynamic range is related to thesignal-to-noise ratio (SNR).

These problems noted above are particularly endemic to CMOS imagesensors, which have a lower light sensitivity and narrower dynamic rangethan charge coupled devices (CCD's).

Prior attempts to solve these problems have included increasing theexposure time using an electronic shutter. However, this method cannotbe applied to pinned photodiodes and may result in movement artifacts.

Another method utilizes frame addition using multiple frames that arecaptured of the same image. However, this technique relies upon oneframe for middle and low light levels and another frame for high lightlevels. This is accomplished using different exposure times for eachframe. Nevertheless, it is difficult to add frames that have differentexposure times and this degrades picture quality.

Another method uses an image sensor that has pixels with two differentfill factors. This type of sensor though has poor resolution for lowlight level images. Finally, another method uses a high frame drivingrate that includes analog-to-digital (ADC) circuit in each pixel. Thiscauses a large pixel size. Further, resolution is limited since movementartifacts may appear.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference numerals refer to like parts throughoutthe various views of the non-limiting and non-exhaustive embodiments ofthe present invention, and wherein:

FIG. 1 is a schematic diagram of a CMOS image sensor.

FIG. 2 illustrates four frames taken by the image sensor and used toform the composite image.

FIG. 3 is a flow diagram of the method of the present invention.

FIG. 4 is a schematic diagram of an image sensor of the presentinvention.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

In the following description, numerous specific details are provided,such as the identification of various system components, to provide athorough understanding of embodiments of the invention. One skilled inthe art will recognize, however, that the invention can be practicedwithout one or more of the specific details, or with other methods,components, materials, etc. In still other instances, well-knownstructures, materials, or operations are not shown or described indetail to avoid obscuring aspects of various embodiments of theinvention.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearance of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

With reference to FIG. 1, an architecture for a CMOS imaging array 101includes a rectangular matrix of pixels 103. The number of pixels in thehorizontal or x-direction, and the number of pixels in the vertical ory-direction, constitutes the resolution of the imaging array 101. Eachof the pixels 103 in a vertical column routes its signal to a singlecharge amplifier 105.

The retrieval of information from the pixels 103 follows the well-knownraster scanning technique. In particular, a row of pixels 103 is scannedsequentially from left to right. The next row is then scanned in thismanner until all rows have been scanned sequentially from top to bottom.At the end of each complete scan of the entire array 101, a verticalblanking period of predetermined time occurs until the raster scanningpattern is repeated. This type of scanning follows the NTSC scanningscheme. However, for other applications, different readout protocols maybe followed. Control circuitry of conventional design is operative tosequentially read the pixels 103 in this manner.

As each pixel is scanned, the signal from that pixel is provided to thecharge amplifier 105 for that column. Thus, the charge amplifiers 105receive signals sequentially. The sequential signals from the chargeamplifiers 105 are then forwarded to a second-stage amplifier 107, whichamplifies the signals so that they may be farther processed. In variousembodiments, either the column amplifier or the second-stage amplifiermay be eliminated. The gain of the column amplifier or the second-stageamplifier may be selectively increased or decreased as the need arises.The readout of the pixels 103, the amplifiers 105 and 107, and thecalculations and processes described below may be carried out byprocessor 111.

As noted above, one important function of the image sensor is to be ableto compensate for low light conditions or conditions that include highlight and low light sections. An example would be taking an imageoutdoors in a sunny day, but with the object of the image in a shadow.

The present invention increases dynamic range and minimizes movementartifacts by using a “frame addition” technique. This involvesincreasing the frame rate of the image sensor and combining (such as byadding) successive frames together to obtain a composite image. As usedherein, the term frame is a captured image that is to be used internallyby the image sensor for forming a composite image.

As one example, a typical image capture rate is 30 frames per second.This corresponds to a 1/30 second exposure time for each pixel. However,if the frame rate is increased to 120 frames per second, then theexposure time is reduced to 1/120 seconds. This smaller amount of timereduces the amount of movement artifacts present in the image. In fact,objects in the image will typically only move a few pixels during a1/120 second exposure time. This enables the detection of motion to bedone precisely.

As seen in FIG. 2, as one example, four successive frames 201 a–d withan object 203 are captured. The frames are captured at a fast exposuretime, such as 1/120 seconds. As a result, there is very little blurringof the objects 203 within each individual frame 201 a–d (referred to asintraframe movement), even with movement of the objects in the frames.However, as seen in FIG. 2, the objects 203 do move significantlybetween frames 201 a–d (referred to as interframe movement). Thus, as aresult of holding instability of the user, the object 203 movesdownwardly from frame 201 a to 201 b. Similarly, the object 203 movesupwardly and to the right from frame 201 b to 201 c. Finally, the object203 moves downwardly and to the left from frame 201 c to 201 d.

As will be detailed below, the four frames 201 a–d are used to form acomposite image 205 that has a substantially higher signal to noiseratio, and thus, higher dynamic range. In one embodiment, the compositeimage 205 is formed from a simple mathematical additional of the signalsfrom each of the frames 201 a–d. In other embodiments, othermathematical or signal processing techniques may be used to form thecomposite image 205.

Further, the example described herein utilizes four frames each having a1/120 second exposure time. However, it can be appreciated that thenumber of frames used to form the composite image 205 and the exposuretime of each frame may be varied, depending upon the application andvarious design tradeoffs.

Importantly, because of the interframe movement, a reference point isidentified for each frame 201 a–d. This is also referred to as thescanning start point 207. The scanning start point 207 is used to aligneach of the frames 201 a–d so as to eliminate the effect of interframemovement. The identification of a reference point can be accomplishedusing any one of a number of techniques, such as the techniquescurrently used for motion stabilization in video recorders. Alternately,other prior art techniques, such as the use of an accelerometer or theuse of motion detection algorithms commonly used in optical mice may beemployed.

FIG. 3 is a flow diagram showing the method of the present invention. Inone embodiment, at box 301, each of the frames 201 a–d are capturedusing the same exposure time. This reduces the difficulty in combiningthe frames in a coherent manner. However, in some other embodiments, theexposure time may be varied in a known and controlled manner.

Next, at box 303, the frames 201 a–d are stored in memory and areanalyzed to determine the relative interframe movement. Once this hasbeen done, then the frames are aligned with each other to ensure thatthe objects 203 of each frame are “atop” each other.

Next, at box 305, the frames 201 a–d are combined with one another toform the composite image 205. The combination may be a simple arithmeticaddition, or alternatively, some weighted function. The addition can bedone by an arithmetic or other type of processor located on the imagesensor. It can be appreciated that a multitude of various known priorart software or hardware can be used to implement the combinationprocess.

In an alternatively embodiment, in order to save on memory requirements,the successive frames are added sequentially into memory in an additivemanner. For example, the first frame can be stored in a frame memory.The second frame is simply just added into the data stored into theframe memory without the need for separate memory. Of course, thescanning start point 207 should be identified so that the addition ofthe second frame to the frame memory is aligned with the first frame.The process is repeated with the third and fourth frames.

FIG. 4 is a CMOS image sensor 401 formed in accordance with the presentinvention. In many respects, the image sensor 401 is substantiallysimilar to the currently available image sensors manufactured byOmniVision Technologies, Inc., the assignee of the present invention.For example, the image sensor includes the imaging array 101, theamplifiers 105, signal processing 111, I/O 403, JPEG compressioncircuitry 405, and other standard components. However, in accordancewith the present invention, the image sensor 401 also includes a framememory 407 and a movement detector 409. The frame memory 407 is used asan accumulation register that accumulates the signals from the variousframes. The movement detector 409 is operative to align the frames 201a–d so that the reference points coincide.

The present invention described above provides a wider dynamic rangethan prior art techniques. By using a higher frame rate, such as 120frames per second, this provides a four times increase in dynamic rangecompared to a frame rate of 30 frames per second. Even if the noisefloor of a CMOS image sensor is twice as large as a comparable CCD imagesensor, this still results in approximately twice the dynamic rangeincrease.

Furthermore, the present invention compensates for holding instabilityof digital still cameras and mobile phones. This allows long exposuretimes and thus operation in low light conditions.

While the invention is described and illustrated here in the context ofa limited number of embodiments, the invention may be embodied in manyforms without departing from the spirit of the essential characteristicsof the invention. The illustrated and described embodiments aretherefore to be considered in all respects as illustrative and notrestrictive. Thus, the scope of the invention is indicated by theappended claims rather than by the foregoing description, and allchanges which come within the meaning and range of equivalency of theclaims are intended to be embraced therein.

1. A method of forming an image using a CMOS image sensor comprising:capturing a plurality of frames using said image sensor, wherein each ofsaid plurality of images are captured using substantially the sameexposure time; identifying a reference point in each of said pluralityof images; aligning said plurality of frames using said reference point;and combining said plurality of frames into said image.
 2. The method ofclaim 1 wherein said combining is an arithmetic combination of likepixels in said plurality of images.
 3. A method of forming a highdynamic range image using a CMOS image sensor comprising: capturing afirst frame using said image sensor; storing said first frame in a framememory; identifying a reference point in said first frame; capturing asecond frame using said image sensor; aligning said second frame to saidfirst using said reference point; and adding said second frame to saidfirst frame in said frame memory, wherein said first and second framesare captured using substantially the same exposure time.
 4. The methodof claim 3 wherein additional frames are captured by said image sensor,aligned using said reference point, and added to said frame memory. 5.The method of claim 3 wherein said adding is an arithmetic combinationof like pixels in said first and second frames.